Gesture recognition device and method for sensing multi-factor assertion

ABSTRACT

A gesture-recognition (GR) device is disclosed that includes a capacitive touch sensor panel and a controller. The capacitive touch sensor panel comprises a plurality of sensing pads arranged in a cylindrical pattern inside a handle of the GR device and detects a multi-factor touch assertion at a set of sensing pads of the plurality of sensing pads. The controller transmits a driving signal to each of the plurality of sensing pads for the detection of the multi-factor touch assertion, generates an assertion signal, determines a signal sequence based on the assertion signal, and converts a current inactive state of the GR device to an active state based on a validation of the determined signal sequence corresponding to the multi-factor touch assertion and an inferred user intent.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application claims priority to, and the benefit from, and isa continuation-in-part of International Application No. PCT/US20/49372,filed on Sep. 4, 2020, and which claims priority to U.S. ProvisionalApplication No. 62/897,220, filed on Sep. 6, 2019. The presentapplication also claims priority to U.S. Provisional Application No.63/250,315, filed on Sep. 30, 2021.

Each of the above referenced patent applications is hereby incorporatedherein by reference in its entirety.

FIELD

The present application relates to machine-human interfaces, and moreparticularly to apparatus, systems and methods for enablinggesture-centric control input for computer processes, and relatedapplications. Further, certain embodiments of the disclosure relate to agesture recognition device and a method for sensing multi-factorassertion.

BACKGROUND

Various user interface devices have been developed for interacting withmachines, especially computers, by detecting movement of a body part orhand-held device.

A first type uses sensors to detect body movement using a wirelesssensor, for example an infrared sensor.

A second type relies on sensing movement of a sensor that is attached toor grasped by the user. For example, pointing devices, e.g., anelectronic mouse, can detect movement in two dimensions, depending onthe model. Smartphones and similar devices include position andorientation sensors that can sense movement of the device for input toany application the device can run. Handheld controllers for virtualreality translate hand movement into virtual hand movement in a virtualspace.

Toy electronic wands that emit light or tactile vibration when graspedor waved about are also available. These toys lack the ability tocontrol external devices. The user receives the mere pleasure ofobserving light or tactile emission from the wand. Further, such toyelectronic wands incorporate a reflector in the tip that reflects thelight back after it is emitted by an external infrared (IR) camera. Suchtoy wands require an external facing camera, such as infrared (IR)camera, that traces the movement of the reflected invisible light torecognize the gestures. Thus, such toy wand is required to be within thesight of the external facing camera. However, such toy wands are morelikely to register false positives due to lack identification ofintentional assertion for the purpose of recognizing gesturesintentionally performed by a user. Further, due to frequent touchsensing periods and faster duty cycles, power consumption of such toywands may be quite high.

It is desirable to develop new methods, apparatus and systems forgesture-centric user interfaces, that enable users to control a varietyof electronic devices or perform a variety of actions in the real worldwith gestures.

SUMMARY

This summary and the following detailed description should beinterpreted as complementary parts of an integrated disclosure, whichparts may include redundant subject matter and/or supplemental subjectmatter. An omission in either section does not indicate priority orrelative importance of any element described in the integratedapplication. Differences between the sections may include supplementaldisclosures of alternative embodiments, additional details, oralternative descriptions of identical embodiments using differentterminology, as should be apparent from the respective disclosures.

In an aspect, a gesture-recognition (GR) device made to be held or wornby a user includes an electronic processor configured by programinstructions in memory to recognize a gesture. As used herein, a“gesture” is a pattern of movements, such as, for example, up-and-down,side-to-side, inward-outward and/or any combination thereof. Themovements may be of a sensor integrated with an associated prop. In anaspect, a movement sensor is attached to (e.g., incorporated into theelectronics of) the GR device. In another aspect, the GR device iscapable of sensing 3-dimensional motion with up to six degrees offreedom (three linear axes, and three rotational axes), plus three axesof geospatial orientation if desired, using the movement sensor (e.g.,an inertial measurement unit (IMU)).

To recognize a gesture, the processor of the gesture-recognition (GR)device detects a pattern of movements, classifies the pattern to a typeor rejects it as unclassifiable, and associates the type (if any) to anelectronic instruction. It may perform any or all these operationslocally or remotely, using a heuristic algorithm, a rules-basedalgorithm, or a combination of heuristic and rules-based algorithm. Inan aspect, the processor may access a library comprising a plurality ofaction identifiers associated with a plurality of gesture types toidentify an action associated with the recognized gesture type from alibrary. As used herein, an “action” includes user-directed changes inmachine states, for example, illuminating a light, extinguishing alight, retrieving content, playing content, jumping ahead or backwardsin content, opening a door, or any of the innumerable things that amachine controlled by an electronic processor can do. In the context ofa data library, an “action identifier” is data that enables the actionto be identified, for example, a pointer, an instruction set or module,or other identification code. In a related aspect, the processor, or aprocessor of an associated interface device, may include instructionsthat enable a user to edit the associations between action identifiersand gesture types.

In another aspect, the processor of the GR device, or another incommunication with it, may send a signal to one or more targetedancillary devices, causing each ancillary device to execute instructionsperforming the identified action. For example, the processor may executeinstructions to perform the identified action that includeelectronically transmitting signals to a second electronic processorlocated in a second device. The second device may be, or may include, atleast one of a light, a television, a projector, a refrigerator, apersonal smart device, an appliance, a virtual reality device, anaugmented reality device, a display device, or a toy.

In related aspects, the gesture recognition device may include a lightemitting device (LED), wherein the action may include altering acharacteristic of light emitted from the LED device, such as, forexample, its color, flashing rate, or intensity. The gesture recognitiondevice may include an inertial measurement unit (IMU) configured todetect gestures in three-dimensional space, including gestures havingsix degrees of freedom (3 linear, 3 rotational) or less, plus three axesof geospatial orientation if desired. The electronic processor isconfigured to recognize the gesture based on signals received from theinertial measurement unit.

In some embodiments, classifying gestures by type and associating theaction identifiers and gesture types may be done remotely, e.g., by aremote server or a mobile device, while characterizing a movementpattern as digital data is done by a processor of device that undergoesthe movement. Thus, the GR device may initiate the first criticalprocess in gesture recognition—converting a gesture into a wireless,machine-readable signal that can be characterized by type—withoutperforming later operations in a chain of causation between a gesture bythe user and an action by a target device. In other embodiments, the GRdevice may perform later operations in the chain of causation, up to butnot including performing the action itself. The GR device may alsoperform local actions, for example, emitting sounds, vibrations, orlights, synchronized to the action performed by the targeted device. Inan aspect, the GR device may perform local actions indicating otheroutcomes, such as a failure to classify a gesture of a recognizabletype, or a failure to communicate an action identifier to a targeteddevice. In addition, the GR device may perform local actions indicatingintermediate states, for example successful input of a gesture to type.

In other aspects, a system for providing a personalized experience mayinclude a central electronic processor at a central location, an edgeelectronic processor near a first location, and a plurality of connecteddevices at the first location, wherein the central processor isconfigured to send instructions to control the plurality of connecteddevices at the first location to create a personalized experience for auser at the first location. The plurality of connected devices mayinclude at least one of a user arrival notification system, a light, amirror, a television, a projector, a virtual reality device, anaugmented reality device, a speaker or a microphone.

The system may further include, in a computer memory, encodedinformation about capabilities of the plurality of connected devices atthe first location. The information about capabilities of the pluralityof connected devices at the first location may be in a databaseaccessible by the central processor. In such embodiments, the centralprocessor is configured to send instructions to control the plurality ofconnected devices at the first location to create a personalizedexperience for a user at the first location based on the capabilities ofthe plurality of connected devices at the first location. As usedherein, a “personalized experience” means sensory output from theconnected devices that is configured based on information defined by orfor an individual user indicative of the user's preferences for thesensory output.

In an alternative, or in addition, the information about capabilities ofthe plurality of connected devices at the first location may be in adatabase accessible by the edge processor. In such embodiments, thecentral processor may be configured to send instructions to control theplurality of connected devices at the first location assuming fullcapabilities of the plurality of connected devices and the edgeprocessor may be configured to receive the instructions and provide apersonalized experience for a user at the first location based on theinstructions and on capabilities of the plurality of connected devicesat the first location to command a personalized experience for a user atthe first location.

In an aspect, a GR device may be, or may be locally connected to, anedge processor of the system. The personalized experience may includecontrolling the plurality of connected devices at the first locationaccording to a gesture-recognition library defined by of for a user ofthe GR device. In an aspect, an edge processor or the central processormay translate between

In accordance with an aspect of the disclosure, the GR device mayinclude a capacitive touch sensor panel comprising a plurality ofsensing pads arranged in a cylindrical pattern inside a handle of the GRdevice. The capacitive touch sensor panel may be configured to detect amulti-factor touch assertion at a set of sensing pads of the pluralityof sensing pads. The GR device may further include a controller coupledto the plurality of sensing pads. The controller may be configured totransmit a driving signal to each of the plurality of sensing pads forthe detection of the multi-factor touch assertion. The controller may befurther configured to generate an assertion signal that corresponds tothe detected multi-factor touch assertion. The controller may be furtherconfigured to determine a signal sequence based on the receivedassertion signal. The controller may be further configured to convert acurrent inactive state of the GR device to an active state based on avalidation of the determined signal sequence corresponding to themulti-factor touch assertion and an inferred user intent.

In accordance with an embodiment, the arrangement of the cylindricalpattern of the plurality of sensing pads may be such that a longitudinalaxis of each sensing pad is arranged orthogonally to a circular axisinside the handle of the GR device.

In accordance with an embodiment, the arrangement of the cylindricalpattern of the plurality of sensing pads may enable a 360-degreecapacitive touch for the multi-factor touch assertion.

In accordance with an embodiment, the controller is further configuredto infer the user intent may be based on the detected multi-factor touchassertion and the determined signal sequence. The user intent may beinferred based on a combination of type of grip technique on the handleand subsequent finger and/or thumb press performed by a user.

In accordance with an embodiment, the assertion signal is generated maybe based on the detected multi-factor touch assertion for a plurality ofgrip techniques. The assertion signal may correspond to one of a firstdetection factor, a second detection factor or an optional thirddetection factor. The first detection factor may be an initial assertionof at least one sensing pad on one side of the handle resulting from anatural or learned grip technique performed by a user. The seconddetection factor may be a subsequent additional assertion of at leastone positionally opposite sensing pad by the thumb or index finger ofthe user. The optional third detection factor may be sensing a threedimensional movement of the GR device during the subsequent additionalassertion.

In accordance with an embodiment, a first grip technique from theplurality of grip techniques for the multi-factor touch assertion maycorrespond to a first touch and continually maintained assertion by asecond digit of hand on at least one sensing pad. The assertion signalmay be generated based on the first grip technique.

In accordance with an embodiment, a second grip technique from theplurality of grip techniques for the multi-factor touch assertion maycorrespond to a first touch and second touch continually maintainedassertions by a first and third digits of hand on at least two sensingpads, and a third touch asserted by a second digit of the hand on aremaining sensing pads between the at least two sensing pads. Theassertion signal may be generated based on the second grip technique.

In accordance with an embodiment, a third grip technique from theplurality of grip techniques for the multi-factor touch assertion maycorrespond to a first touch and continually maintained assertion by afirst digit of hand on at least one sensing pad and a second touchasserted by a second digit of the hand on at least one positionallyopposite sensing pad. The assertion signal may be generated based on thethird grip technique.

In accordance with an embodiment, a fourth grip technique from theplurality of grip techniques for the multi-factor touch assertion maycorrespond to a tap gesture provided by a first or a second digit ofhand on at least one sensing pad. The assertion signal is generatedbased on the fourth grip technique.

In accordance with an embodiment, an accelerometer, coupled with theplurality of sensing pads, may be configured to detect the tap gesture.

In accordance with an embodiment, the controller may be furtherconfigured to change the active state of the GR device to a sleep statein absence of the multi-factor touch assertion at the set of sensingpads for a pre-defined time duration.

In accordance with an embodiment, the conversion of the current state ofthe GR device to the active state may be further based on the drivingsignal received from the controller and a touch gesture received from atleast a digit of hand of a user.

In accordance with an embodiment, a frequency value of the drivingsignal received from the controller has a pre-defined value.

In accordance with an embodiment, a sensing period and a duty cycle ofthe plurality of sensing pads is less than a threshold value.

In accordance with an embodiment, the GR device may be a wirelessinteractive wand, or a smart wand configured to communicate wirelesslyvia a radio frequency (RF) or an infrared (IR) communication mode withother devices by utilizing power generated by a power storage unit.

In accordance with an embodiment, the GR device may be an interactivewand configured to illuminate in a plurality of sections by utilizingpower generated by a power storage unit.

In accordance with an embodiment, the GR device may be an interactivewand configured to generate haptic feedback by utilizing power generatedby a power storage unit.

In accordance with an embodiment, the capacitive touch sensor panel maybe integrated on a flex printed circuit board and wrapped to form acylindrical shape within the GR device.

In accordance with an embodiment, the capacitive touch sensor panel maybe communicatively coupled to a printed circuit board housing thecontroller via a flex connector.

In accordance with another aspect of the disclosure, a capacitive touchsensor panel may include plurality of sensing pads is arranged in acylindrical pattern inside a handle of a GR device, the arrangement ofthe plurality of sensing pads in the cylindrical pattern is such that alongitudinal axis of each sensing pad is arranged orthogonally to acircular axis around the handle of the GR device, the plurality ofsensing pads communicatively coupled to a capacitive touch sensorcontroller. The capacitive touch sensor controller may be configured totransmit a driving signal, received from a controller, to each of theplurality of sensing pads. The capacitive touch sensor controller may befurther configured to detect a multi-factor touch assertion at a set ofsensing pads from the plurality of sensing pads, generate an assertionsignal in response to the detected multi-factor touch assertion, andtransmit the generated assertion signal to the controller. The GR devicemay be activated by the controller based on a validation of a signalsequence determined based on the generated assertion signal and aninferred user intent.

In accordance with an embodiment, the plurality of sensing pads may bearranged in the cylindrical pattern inside the handle such that theplurality of sensing pads is in proximity to hand digits of a user.

In accordance with yet another aspect of the disclosure, a method forsensing multi-factor touch assertion may include transmitting, by acontrol unit, a driving signal to a plurality of sensing pads fordetection of multi-factor touch assertion. The method may furtherinclude activating, by the control unit, the plurality of sensing padsbased on the received driving signal. The method may further includedetecting, by the control unit, multi-factor touch assertion at a set ofsensing pads from the activated plurality of sensing pads. The methodmay further include generating, by the control unit, an assertion signalin response to the detected multi-factor touch assertion, The method mayfurther include determining, by the control unit, a signal sequencebased on the received assertion signal. The method may further includeinferring, by the control unit, a user intent based at least on thedetected multi-factor touch assertion and the determined signalsequence. The method may further include converting, by the controlunit, a current inactive state of the GR device to an active state basedon a validation of the determined signal sequence corresponding to themulti-factor touch assertion and the inferred user intent.

In accordance with an embodiment, the assertion signal may be generatedbased on the detected multi-factor touch assertion for a plurality ofgrip techniques. The assertion signal may be generated based on one of afirst grip technique, a second grip technique, a third grip technique,or a fourth grip technique from the plurality of grip techniques.

As used herein, a “client device” or “device” includes at least acomputer processor coupled to a memory and to one or more ports,including at least one input port and at least one output port (e.g., adesktop computer, laptop computer, tablet computer, smartphone, PDA,etc.), including accessories such as wands, rings, and staffs soequipped. A computer processor may include, for example, amicroprocessor, microcontroller, system on a chip, or other processingcircuit. As used herein, a “processor” means a computer processor.

To the accomplishment of the foregoing and related ends, one or moreexamples comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative aspectsand are indicative of but a few of the various ways in which theprinciples of the examples may be employed. Other advantages and novelfeatures will become apparent from the following detailed descriptionwhen considered in conjunction with the drawings and the disclosedexamples, which encompass all such aspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify like elements correspondingly throughout thespecification and drawings.

FIG. 1 is a diagram illustrating a GR device and components thereof.

FIG. 2 is a diagram illustrating use cases for a GR device in aconnected environment.

FIG. 3 is a diagram illustrating a smart retail environment embodying anaspect of a system for providing a personalized experience.

FIG. 4 is a system diagram illustrating hardware components of a systemfor providing a personalized experience, including a GR device.

FIG. 5 is a block diagram illustrating programmable components of asystem for providing a personalized experience, including a GR device.

FIG. 6 is a flow chart illustrating aspects of gesture recognitiontraining for use in or with a GR device.

FIG. 7 is a flow chart illustrating aspects of characterizing a datasignature for use in or with a GR device.

FIG. 8A-B are flow charts illustrating aspects of accessing a library ofgesture-action associations and recognizing a gesture type.

FIG. 9 is a flow chart illustrating aspects of performing an actionspecified by a gesture.

FIG. 10 is a flow chart illustrating a process for building or adding apersonalized gesture library for use in or with a GR device.

FIG. 11 is a flow chart illustrating a method for controlling a GRdevice to provide a gesture-centric user interface for controlling oneor more connected devices.

FIG. 12A is a diagram that illustrates an external view of an exemplaryGR device.

FIG. 12B is a diagram that illustrates an unassembled view of theinternal circuitry, of the exemplary GR device of FIG. 12A with acapacitive touch sensor panel for detecting a multi-factor touchassertion.

FIG. 12C is a diagram that illustrates an assembled view of the internalcircuitry, of the exemplary GR device of FIG. 12A with a capacitivetouch sensor panel for detecting a multi-factor touch assertion.

FIG. 12D is a circuit diagram of I/O drive of the capacitive touchsensor panel in the exemplary GR device.

FIG. 13 is a flow chart illustrating a method for sensing multi-factortouch assertion by a GR device.

FIGS. 14A to 14D are diagrams that collectively illustrate plurality ofgrip techniques of a GR device of FIG. 12A for the multi-factor touchassertion.

FIG. 15 is a conceptual block diagram illustrating components of anapparatus or system for providing a gesture-centric user interface forcontrolling one or more connected devices.

DETAILED DESCRIPTION

Various aspects are now described with reference to the drawings. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth to provide a thorough understanding of one or moreaspects. It may be evident, however, that the various aspects may bepracticed without these specific details. In other instances, well-knownstructures and devices are represented in block diagrams relating whatis known to novel aspects of the present disclosure.

Referring to FIG. 1 , gesture-recognition (GR) device 100 may be usedwith one or more connected devices to provide enhanced experience athome, retail locations, theme parks, theaters and other locations. TheGR device may include a controller 106 (e.g., a Raspberry Pi OW)configured to perform operations of the GR device 100, includingdetecting different gestures formed by motion of the GR device andtriggering correlated actions to be performed by connected devices, forexample, devices 201-213 shown in FIG. 2 , in response to each gesture.In accordance with various embodiments, the controller 106 may beinterchangeably referred to as a microcontroller (MCU) or other relatedterminology, without any deviating from the scope of the disclosure. Theconnected devices 201-213 to be individually described later may eachperform an action determined based on a gesture performed by user 99holding or wearing the GR device 100, to provide an enhancedpersonalized experience. The user moves the GR device through a patternof motion, and with or without a verbal command, one of the connecteddevices performs a desired action. A good deal of hardware and softwaremakes this appearance of magic possible.

Referring again to FIG. 1 , an embodiment of the GR device 100 mayinclude two principal components, a base unit 102 and an elongate unit104. In some embodiments, the base unit 102 and the elongate unit 104may be configured as a single piece. For a GR device 100 styled as awand, the base unit 102 may be configured to be grasped by the user'shand, while the elongate unit 104 may be configured to have a formfactor of a wand tip that extends outward from the base unit 102 andprovides the wand's overall appearance. In some embodiments, theextendable unit 104 may be configured to be removably attached to thebase unit 102. In some embodiments, the base unit 102 may be permanentlyfixed to the elongate unit 104. In some implementations, all theelectronics may be placed in the base unit 102 while the elongate unit104 can be devoid of any electronics. In some other implementations, afirst portion of the electronics may be disposed in the base unit 102and a second portion of the electronics may be disposed in the elongateunit 104. For example, heavier, more expensive electronic components maybe placed in the base unit 102 while relatively inexpensive electroniccomponents may be placed in the elongate unit 104. The elongate unit 104may be provided with different appearances and capabilities to suitusers' needs.

To provide interchangeability, a removable fastener 118 with electricalcontacts 115, 117, 119 may be used to couple the base and elongate units102, 104. While an internally-threaded coupling is shown, othercouplings may also be suitable, for example, an externally threadedplug-and-socket, a threadless plug-and-socket with or without a lockingfeature, and so forth. Since the GR device is designed to be movedrapidly around, a locking feature is advantageous to prevent undesireddecoupling.

Capabilities of the GR device 100 may be limited or enhanced dependingon an identity of a user of the GR device 100 or the elongate unit 104.For example, the elongate unit 104 may include a radio-frequencyidentification device (RFID) 168 or other identification device, and agesture recognition and control system of the GR device 100 may beconfigured to work differently depending on the identity of the elongateunit. For example, special commands may be “unlocked” for certainidentifiers. In home settings, commands may be refused except frommovements of GR devices that include identifiers registered for ahousehold. As another example, special commands may be unlockeddepending on the identity of a user. Information regarding the identityof a user may be communicated to the GR device 100 via one or moreconnected devices associated with the GR device 100.

While a wand is illustrated, it should be appreciated that a GR device100 may have any suitable form factor for being held or worn by a userand carried by movements of a user's extremities. For example, a GRdevice 100 may be styled as a walking staff, a light stick, a ring, abody ornament, a glove, a bracelet, or any article capable of being heldand moved through the air by a user. For further example, in a ring,bracelet, or similar jewelry, the base unit 102 may be contained in thebody of the jewelry while the elongate unity 104 may be styled as a gemor ornament.

Ornamental features aside, operation of the GR device 100 depends on itsinternal circuitry and elements in its wireless network. The internalcircuitry of the GR device 100 may include a controller 106 coupled toan inertial measurement unit (IMU) 108, to a power storage unit 110(e.g., a battery), and to an input-output and power connector 112 (e.g.,a Universal Serial Bus (USB) port). Optionally, the controller may becoupled to one or more auxiliary devices 114, 116, described in moredetail herein below, and to electronics in the elongate unit 104, forexample, one or more light-emitting devices (LEDs) 120 and accompanyingcontroller 160, if any.

The IMU 108 (e.g., sensor BN0055 from Bosch) may include one or moreaccelerometers 172 for acceleration detection, one or more gyroscopes174 for force and movement detection, and a magnetometer for geographicorientation. The GR device mat include one or more IMUs 108, which maybe in the base unit 102, the elongate unit 104, or in both the primaryand elongate units. The IMU may include a processor 170 that determinesfrom sensor data magnitude and direction of motion in up to threespatial axes, three rotational axes, and three geospatial orientationaxes, or other useful metric for determining pattern of movement and theassociated gestures, using any desired coordinate system (e.g.,Cartesian or spherical) and any useful sampling interval, such as forexample, 0.1 to 100 milliseconds. The IMU 108 may output other usefulinformation, for example, its geospatial orientation. When the wand 100is moved in space to perform a gesture, data from the accelerometer, thegyroscope and/or the magnetometer of the IMU 108 is processed by theprocessor 130 to detect the pattern of movements, identify the gestureand associate it with an action to be performed by a connected device.As described in more details herein, the processor 130 may access localor remote data structures and servers to complete identification of thegesture and selection of an appropriate associated action. The processor130 may execute the identified action (e.g., illuminating LED 120 with aspecific color or emitting a predetermined sound from an audiotransducer 138, 166), cause instructions to be sent to a connecteddevice, or both.

The controller 106 may include a processor 130 coupled to a randomaccess memory (RAM) 156 holding program instructions and data for rapidexecution or processing by the processor during operation. When theapparatus 100 is powered off or in an inactive state, programinstructions and data may be stored in a long-term memory, for example,a non-volatile magnetic, optical, or electronic memory storage device157. Either or both of the RAM 156 or the storage device 157 maycomprise a non-transitory computer-readable medium holding programinstructions, that when executed by the processor 130 cause the GRdevice 100 to perform operations as described herein for gesturerecognition and control, alone, or in combination with one or moreadditional processors. The one or more additional processors may becoupled locally to the processor 130, remotely via a wirelessconnection, or both. Program instructions may be written in any suitablehigh-level language, for example, C, C++, C#, JavaScript or Java™, andcompiled to produce machine-language code for execution by theprocessor. Program instructions may be grouped into functional modules,to facilitate coding efficiency and comprehensibility. It should beappreciated that such modules, even if discernable as divisions orgrouping in source code, are not necessarily distinguishable as separatecode blocks in machine-level coding. Code bundles directed toward aspecific function may be considered to comprise a module, regardless ofwhether or not machine code on the bundle can be executed independentlyof other machine code. In other words, the modules may be high-levelmodules only.

To assist with personalization and operation of the GR device 100, thecontroller 106 may be attached to various other input and outputdevices, arranged with it on a module circuit board 107 and/or elsewherein the GR device 100, for example arranged onto a single circuit boardwithin the GR device 100. For example, the controller 106 may beattached to a microphone 132 to receive voice commands, and an audiotransducer (e.g., a speaker or piezoelectric device) for audio output.The controller 106 may include a graphics or text processing unit 142providing a signal for controlling output of an electronic text orgraphic display 101. The display 101 may be integrated with the baseunit 102 or may be provided in an auxiliary device (e.g., a smartphone)that couples to the processor 130 via one or more wireless transceivers146, 150. The transceivers 146, 150 may support one or more protocols,for example 5G, Bluetooth, NFC or WiFi. The display 101 may displaytext, picture or video animations based on a gesture input. Theprocessor 130 and memory 156 may be configured with one or more modulesfor speech to text conversion, gesture to speech conversion, or gestureto text conversion. In an alternative, the display 101 may be used as auser interface for configuring the GR device. The controller may includea motion controller 152 for driving an electric motor of a rotational,vibrational, or pulsating feature installed near an externally-facingsurface, e.g., at block 116 in base unit 102. The controller 106 mayinclude an ambient light sensor 134 to detect ambient light levels, askin conductance sensor 136 for biometric sensing, a proximity detector148 to detect when the device is in proximity of other connecteddevices, an RFID sensor 140 for reading identifiers from an RFID device168 of the elongate unit 104 or other device, a particle (e.g. smoke orvapor) emitter for special effects, and a geolocating device (GPS)receiver 154.

It may be advantageous to locate certain sensors or output devices at,on or near an external surface of the GR device 100, for example atblock 114 (Auxiliary 1). Suitable devices located may include, forexample, a biometric sensor such as an electrode array to detect heartrate of the user, a thermistor to detect skin temperature, the skinconductance sensor 136, the particle emitter 144, a scent detector oremitter, a fingerprint reader for user authentication, and/or a heatingelement to enable the device to heat or cool based on gesture input.

In various embodiments, an ancillary controller 160 may be used toaugment capabilities of the primary controller. As illustrated, theancillary controller includes a processor 162 and memory 164 holdingprogram instructions for controlling one or more LEDs 120 and anancillary audio transducer 166. The controller may include an ID device168 for positively identifying the model and serial number of theelongate unit 104, in support of interchangeable variety in programfunctions and system security. The controller 160 may include any one ormore devices and sensors described in connection with 106, for example,a second IMU. Separated IMUs in the base and elongate units 102, 104 maybe useful for more sophisticated gesture recognition, especially forflexible GR devices. For example, a glove configured as a GR device withmultiple IMUs to capture motion of each separately movable part of ahand may be used to detect the full complexity of human language (e.g.,American Sign Language). In embodiments, the 162 and memory 164 may beomitted, and local processing may be implemented only in the base unit102, e.g., processor 130.

Before describing more technical features of the GR device and relatedsystems and methods, applications for gesture recognition by a GR devicewill be described, in context of a connected environment 200 as shown inFIG. 2 .

In an aspect, a GR device 100 may be configured as an Internet of Things(IoT) device, for example, a camera 101 or a light control module 203may be controlled by the GR device 100. The GR device 100 may beconfigured to activate other connected devices based on gestures and/orvoice commands of a user 99. For example, gestures can be used controllights, fans, air conditioners, toasters, refrigerators, doors, garagedoors, cars, vacuum cleaners and other home appliances. In someimplementations, the CR device can directly interact with another IoTdevice. However, in other implementations, the GR device may beconfigured to interact with other IoT devices through an intermediarysystem such as, for example, Google Home, Alexa, or other IoT hub.

In another application, a GR device may be configured to interact with avariety of toys 211, 213 (e.g., balls, cars, vehicles, dolls, robots,etc.) For example, gestures by the user 99 holding the GR device 100 maybe used to control movement of a vehicle, a ball, or a figure (e.g.,doll or robot). The toys may be configured as IoT devices, or aswireless devices configured for direct connection to the GR device orindirect connection through an auxiliary device (s).

In other applications, a GR device 100 may be used to provide a varietyof augmented reality (AR) or virtual reality (VR) experiences 209.Gestures may be used to control virtual objects in a VR environment, forexample, by communicating gestures detected by the GR device 100 to a VRdevice worn by the user to control one or more virtual objects. Gesturesmay also be used to control virtual objects in an AR environment. Inthis scenario, one or more virtual objects can be overlaid over objectsin the real world (e.g., a virtual ball/feather is placed on a table inthe real world). The gestures detected by the GR device 100 may becommunicated to the AR device 209 worn by the user to control one ormore virtual objects.

A GR device may be used to enhance entertainment presented over atelevision 205, notepad computer 209, projector 207, or other contentdelivery device. For example, gestures made with the GR device may beused to interact with real and/or virtual objects or projected images tounlock additional content and/or bonus features (e.g., additionalscenes, making of the scene, etc.) in an entertainment setting (e.g., ata theater/a theme park/cruise ship/some other entertainment setting).Gestures with the GR device may be used to enhance the experience ofwatching a movie or playing a game. For example, gestures can be used toadd content to a scene when watching a movie or a game. As anotherexample, gestures can be used to control the narrative of a movie. Thedevice can light-up, vibrate and/or buzz at climactic movements whilewatching a movie or a show.

In the area of retail sales, a GR device 100 may be configured toidentify an item selected by the customer using gesture control. If theitem is wearable (e.g., clothing, footwear, headwear, accessory, or thelike) a retail experience system in communication with the GR device 100may be further configured to display or project an image of the customerwearing the selected item in the customer's size based, on a gesturemade by the GR device 100. The image can be displayed on a displaydevice (e.g., a smart phone, a smart mirror, a computer, a smart pad,etc.) or projected in the ambient environment. The customer can usegesture control to change the size and/or color of the selected item ofclothing/shoe. The customer can perform another gesture to buy theselected item of clothing/shoe. The selected item of clothing/shoe canbe delivered to a preferred location of the customer.

For social applications, a GR device 100 may be personalized to theuser. For example, a GR device 100 may be configured to recognize theuser's biometric/voice and retrieve personal information associated withuser (e.g., name, birthday, affiliations, preferences, and so forth). Asanother example, a GR device 100 can provide a unique user identifier toa user recognition system which can further retrieve personalinformation associated with user (e.g., name, birthday, affiliations,preferences, and so forth). The retrieved personal information can beused to recognize the user at theme parks, retail locations, theaters,or other venues; keep track of rewards, offer redemption of rewards,provide personalized service, customize offers, or other communicationactions. Personal information retrieved with permission via a user's GRdevice can be used to greet the user upon entry into a space, alter thecolors or other decorations of the space to reflect the user'saffiliations/preferences. The retrieved personal information can alsoinclude a list of the user's friends or other people associated with oneor more social groups that the user belongs to. The GR device may beconfigured to receive information of the user's friends or other peopleassociated with one or more social groups that the user belongs to inthe user's vicinity and alert the user to their presence to facilitatesocial interaction. Further to enhance social communication, a GR devicemay be equipped with gesture to speech conversion or gesture to textconversion capabilities. Accordingly, a GR device may facilitatecommunication between individuals who don't speak the same language.These capabilities can also be beneficial to individuals withdisabilities. As a geospatial locating device, a GR device may be usedas, or as part of, a navigation instrument capable of providing turn byturn directions from origin to destination to a user.

A GR device may be used to enhance a consumer experience at a retaillocation and encourage sales. In an illustrative application, a user isnotified via an application (e.g., a magic/fantasy app) on a personalsmart device (e.g., an Android device, iPhone, etc.) that he/she is noweligible to purchase an item (e.g., a special robe, a special toy, anaccessory, etc.) from a retail location. The user's arrival at theretail location may be communicated by the application on the user'spersonal smart device to a user arrival notification system located inthe vicinity of entrance of the retail location. Thus, when the userarrives at the retail location, a retail greeting system or theapplication may welcome the user with a personalized greeting. Thepersonalized greeting may include, for example, the user's name, thename of the item they wish to purchase, the area of the retail locationwhere the item is stored, and other pertinent information.

The user arrival notification system may be a part of a smart retailinfrastructure 300, illustrated in FIG. 3 . The user arrivalnotification system can include a communication system configured toreceive information from the application on the user's personal smartdevice. An implementation of a smart retail infrastructure is describedbelow. In some implementations, the smart retail infrastructure mayreceive some or all the relevant user information from the applicationon the user's personal smart device at the time of receiving the user'sarrival information. For example, the smart retail infrastructure mayreceive some or all the relevant user information from one or moredatabases (e.g., databases located in the cloud). The relevant userinformation may be retrieved from the cloud based on informationreceived from the application on the user's personal smart device.

At the retail location, the user may be provided with a generic wand(e.g. GR device 100) if the user doesn't already have a personal wand oftheir own. The wand can be configured to guide the user to the locationwhere the item to purchase is located. For example, the wand may vibrateor illuminate when the user points the wand in the direction of thelocation of the item to purchase. The wand may be used in conjunctionwith a map of the retail location that is displayed by the applicationon the user's personal smart device. The wand may be configured toprovide turn by turn directions to the user in the retail location toguide the user to the location of the item to purchase. The wand may beconfigured to buzz, vibrate, become hot/cold, light-up and/or point toguide the user to the location of the item to purchase.

Upon reaching the location of the item to purchase, the smart retailinfrastructure may prompt the user try on the item for size when theitem to purchase is an article of clothing, footwear, headgear, eyewear,jewelry or some other wearable accessory. The smart retailinfrastructure may prompt the user via the user's personal smart device,via speakers deployed in the retail location and/or via speaker of thewand. In some implementations, the user may try on the item virtually.

In an aspect, the smart retail system may include a smart mirror 312,comprising a computer display coupled to a camera 316, and a videoserver 314. The smart mirror at the retail location may display an image320 of the user 318 wearing the article of clothing, footwear, headgear,eyewear, jewelry or some other accessory 322 (headgear only shown). Inanother aspect, the smart mirror can be configured to detect and notethe user's emotional response to the appearance in the smart mirror, forexample the user's facial expression indicating satisfaction ordispleasure. In some implementations, the smart mirror and/or theapplication on the user's personal smart device may offer size or colorrecommendations to the user. The size or color recommendations may bemade based on the user's preference, the detected emotional responsewith the item being tried on and other considerations. A display oraudio output of the smart mirror 312 or the application on the user'spersonal smart device may request the user to confirm the color and/orsize of the item and perform a predetermined purchasing gesture if theuser wants to purchase the item. The purchasing gesture may be detectedby the GR wand and transmitted to the smart retail infrastructuredirectly or via the user's personal smart device. Upon receivingnotification of the purchase, the smart retail infrastructure may beconfigured to trigger congratulatory messages (e.g., triggering a lightshow in the vicinity of the user, causing the wand to lightup/buzz/vibrate, issue verbal confirmation of the purchase, etc.) toconfirm the purchase. The user may be notified (e.g., via the smartpersonal device) that the item will be delivered to a preferred deliverylocation.

The application on the user's personal smart device may be configured tonotify the user regarding the presence of friends and/or other membersin the user's social groups in the retail location. The application onthe user's personal smart device may display the location of the user'sfriends and/or other members in the user's social groups on a map. Thewand in conjunction with the application on the user's personal smartdevice may guide the user to the location of the user's friends and/orother members in the user's social groups. The map of the retaillocation may reveal hidden places within the store location which areaccessible to the user and his friends. Special items and/or discountsmay be available to the user and his friends when the access the hiddenplaces through the map.

The retail location may include other activities to facilitate socialinteraction, such as, for example, photobooth, food stalls, etc. Furtherdetails of a smart retail infrastructure 300 may include a plurality ofretail locations 302, 304, and 306 connected to the smart retailinfrastructure 300. Each retail location 302, 304, and 306 may beassociated with a unique store identifier. Each retail location mayinclude a plurality of connected devices, such as, for example,communication devices at the entrance and various other locations in theretail location, lights, projectors, televisions, speakers, microphones,or other connected devices. The plurality of connected devices in eachretail location are a part of the smart retail infrastructure 300. Thesmart retail infrastructure can interface with other cloudinfrastructures 308 and 310.

The smart retail infrastructure 300 may include information regardingthe capabilities of the various connected devices in each retaillocation. Accordingly, the smart retail infrastructure can customize theuser experience in each retail location based on the capabilities of thevarious connected devices in each retail location.

For example, if a retail location 306 does not have a smart mirror, thenthe smart retail infrastructure may prompt the user to use his/herpersonal smart device to virtually try on the item to purchase. Asanother example, if the lights in the retail location 304 are notcapable of changing colors, then the smart retail infrastructure may notuse the light effects in creating user's retail experience.

In some implementations, each retail location may be provided with edgecomputing device or server 330. In such implementations, thecapabilities of the various connected devices in the retail location maybe stored at the edge of the smart retail infrastructure within the edgecomputing device 330. A central processor 340 of the smart retailinfrastructure may create a user experience that is common to all theretail location assuming full capabilities of the various connecteddevices. Individual edge computing devices may tailor the userexperience for the individual retail location based on the capabilitiesof the connected devices.

FIG. 4 shows additional components and aspects of a gesture recognitionsystem 400 for use with applications described herein, for interactingwith a GR device 402, which may be the same as or similar to the GRdevice 100 previously described. The system may include a smartphone 404with touch screen display 405 in wireless communication with the GRdevice 402. The GR device 402 may communicate with local connecteddevices, e.g., LAN client device 410, via a router 430. The GR devicemay communicate with a short-range wireless (e.g., Bluetooth) clientdevice 408 via a peer-to-peer wireless link. The GR device 402 maycommunicate with wide area network (WAN) IoT clients 418, 420 via a hubserver 416 (or without the hub, as applicable), WAN 414 and router 430or wireless access point 412. Likewise, the GR device 402 may connectwith one or more remote servers 422, that may provide resources forgesture recognition, for example, library data, or code execution forgesture recognition or gesture recognition training. For example, aremote server 422 may classify gestures by type and associate the actionidentifiers and gesture types, while characterizing a movement patternas digital data is done by a processor of the GR device. In variousimplementations, the GR device may initiate the first step in a gesturerecognition process—converting a gesture into a wireless,machine-readable signal that can be characterized by type—withoutperforming later operations in a chain of causation between a gesture bythe user and an action by a target device. In other embodiments, the GRdevice may perform later operations in the chain of causation.

FIG. 5 shows programmable components of a GR system 500 for providing apersonalized experience, including a GR device. Block 502 encompassescritical components of a GR device for local execution. Dashed block 504encompasses components that may be executed by a remote server, by theGR device, or both. Block 506 encompasses components of a connecteddevice that performs an action determine by the GR system 500, forexample connected clients as described in connection with FIG. 2 .Gesture sensing 508 is performed locally by sensors and at least onesignal processor of the GR device, as the user moves the GR device inspace. As used herein, gesture sensing can refer to the detection ofpattern of movements. The GR device and remote server, if any, mayinclude a communication module 510 for communicating data andinstructions with each other and with the connected client 506 via itscommunication module 520, which may be, or may include, for example, astandard IoT interface. The GR device 502 and/or remote server 504 mayinclude a gesture recognition module 512 that classifies certainpatterns of movement into specific categories, also called types orgestures. The GR device 502 and/or remote server 504 may further includea command library or data structure module 516 that associates gestureswith action identifiers (e.g., commands).

The GR device 502 and/or remote server 504 may further include atraining module 514 for configuring new patterns of movement as gesturesto be recognized by the system 500. Thus, a user may configure their owngestures and expand their gesture libraries. The GR device 502 and/orremote server 504 may further include an administration and managementmodule 518 for adding, deleting, and editing entries in their commandlibrary. Thus, a user or administrator may manage and alter librarycontent for changing circumstances and needs.

A client device 506, also called a target or target device, need only becapable of receiving a command via a communications module 520,processing the command signal by an information processing (includingcommand handling) module 522, and controlling its output accordingly viaan output control module 524. Communications protocols used by theclient 506 may be standard protocols, e.g. IoT, Bluetooth, so connectionwith any device capable of connecting via a common protocol is possible.

Before a GR system (e.g., system 500) can recognize a pattern ofmovements, it may need to be programmed or trained to do so. Rules-basedalgorithms for pattern recognition may be programmed manually orsemi-automatically, while heuristic algorithms (e.g., neural networks)may be trained using training sets. In both cases, an envelope for eachgesture is defined. Gestures that fall within the envelope areclassified (recognized) while those that fall outside the envelope arenot classified (unrecognized). FIG. 6 shows aspects of gesturerecognition training method 600 for use in or with a GR device and/orsystem 100, 402, 500. Gestures may be personalized for each user or usercohort, standardized for all users, or defined by both standard andpersonalized factors.

At 602, a processor initializes a training session, for example, inresponse to user or system input requesting training for a new orexisting gesture. At 604, the processor may identify the user of the GRdevice, which may determine which gesture library the gesture belongsto. At 608, the processor may initiate a sampling session, for example,immediately after causing the GR device to emit an audible toneindicating training is to begin. At 610, the processor records motionsensor data for an interval of time, or until motion ceases, dependingon the type of gesture. At 612, the processor determines whether anumber of recorded samples ‘N’ is greater or equal to a minimum numberof samples. If N is less than a minimum needed to characterize a gestureenvelope, the processor reverts to record another sample at 608. If N isnot less than a minimum, then the processor at 614 determines whethervariance between recorded samples is less than a threshold of maximumacceptable variability. If variability is too high and the number ofsamples recorded exceeds a maximum number at 616, the training sessionfails at 618. If variability is too high and the number of samplesrecorded does not exceed the maximum, then the processor reverts torecord another sample at 608.

If variability is within acceptable limits at 614, then the processorcharacterizes the sample set for the gesture at 620. For a rules-basedrecognition algorithm, a sample may be characterized using statisticaltools, for example, mean and standard deviation, in comparing motionvalues across comparable intervals of time. For heuristic algorithms, aneural network or other heuristic process receives feedback from theuser regarding acceptable and unacceptable sample gestures until it canaccurately predict whether a motion pattern qualifies as a gesture.

At 622, the processor relates the data characterizing the gestureenvelope (e.g., statistical ranges or parameters of a heuristic machine)to the identifier determined at 606 in computer memory, for example, ina library database. At 624, if the user wishes to train the system foranother gesture, the processor reverts to block 606 for a newidentifier. Otherwise, the processor completes the session at 624, forexample by signaling the user and/or other devices in the system thatthe training session is complete.

FIG. 7 shows aspects of a method 700 for characterizing a data signaturefor use in or with a GR device. The method may be used whenever desiredto receive and recognize gesture input for applications as describedherein. At 702, the processor waits for gesture input to begin. To avoidwasting processor resources, a user may deactivate the GR device'ssensing capability when not needed, to prevent continual processing ofrandom movement data. Thus, a trigger for gesture sensing may include amanual activation of the sensing function coupled with movement of theGR device. Manual activation may include, for example, receiving aspoken command, e.g., “abracadabra!” from the user. At 704, theprocessor waits until a trigger is received. Once the trigger isreceived, at 705 the processor receives 3D motion data from one or moreIMU sensors. The data may include 3 spatial, 3 rotational, and 3geospatial orientation axes as previously described, or some lessersubset of these 9 axes.

At 710, the processor determines whether any auxiliary data (e.g., averbal command, or other input) is to be included as input to definingthe gesture signature. This may be determined, for example, based onuser preferences or other definition of data making up gesture input.Auxiliary data from a local microphone 712 may be used to supply averbal component, such as a word or sound that included as part of thegesture. A location sensor 714 or example a GPS sensor, may be used toprovide location data to constrain operation of the gesture to thepresent location. A network sensor 716 may similarly be used to providenetwork address data to constrain operation of the gesture to definitenodes of a network. Gesture definition is not limited by these examples.At 718, the processor receives the auxiliary data contemporaneously withreceiving the motion data 706, or a short interval before or afterwards.At 720, the processor applies filers and transformations (e.g., Fouriertransforms) to efficiently encode the gesture data for laterrecognition. An encoded gesture may be referred to herein as a“signature” or “gesture signature.” At 722, the processor outputs thesignature for downstream processing.

FIG. 8A-B show aspects of a method 800 for accessing a library ofgesture-action associations and a related method 804 for recognizing agesture type. At 802, a processor of a GR device or system receives agesture signature. At 804, the processor classifies the character as atype, or as unclassifiable. At 806, the processor queries the user'sgesture library by gesture type. If the gesture type is in the libraryat 808, the processor returns at 812 an identifier for a target clientor clients and an identifier for at least one action associated with thegesture type, for use in controlling the targeted client or clients. Ifthe gesture type does not exist in the library, or if the gesture is notclassified, then the processor may provide a failure signal at 812.

FIG. 8B shows further details of gesture classification 804. At 850, theprocessor applies one or both of a heuristic or rules-basedclassification engine to the gesture signature. If using a heuristicalgorithm, the processor may retrieve parameters for a heuristic enginetrained on the user's gestures, populate the engine with the parameters,and process the signature using the populated engine. For a rules-basedalgorithm, the processor may select a class with the best fit acrossmultiple measures of the movement (e.g., velocity, direction,acceleration, rotation, location) for each interval of time, anddetermine, for the best-fitting class, whether the fit satisfies aminimum threshold of similarity. At 852, if the signature fits within atype, the processor may pass the type identifier; otherwise, it may passa failure signal at 854.

FIG. 9 shows aspects of a method 900 for performing an action specifiedby a gesture. At 902, the processor of a GR device or system may receivea target identifier and an action identifier. At 904, the processorqueries the identified target using the action identifier, according toa query protocol for the target. At 906, the processor determines, basedon a query response, whether the target is ready to perform theidentified action. If the target is not ready, the processor maydetermine if an alternative or additional target is available at 914. Ifan additional or alternative target is available, the processor mayrevert to block 904, query target. If no other target is available, theprocessor may provide a fail signal at 918 and revert to 902 for thenext requested action. If the target is ready at 906, the processor mayrequest that the target perform the action at 908. At 910, the processorconfirms that the action is performed, for example by receiving a signalfrom the target, or sensing a change in the environment caused by theaction. If the performance is confirmed at 910, the GR device or systemmay provide an acknowledgement signal to the user, target, and/oradministrative component of the GR system, and revert to block 902 forthe next action. If the performance is not confirmed, the processor mayrevert directly to block 902.

FIG. 10 shows a process 1000 for building or adding a personalizedgesture library for use in or with a GR device, such as may be performedby a administrative component of a GR system. At 1002, a processor of aGR device or system may authorize a user to edit a specified library ofassociations between action identifiers and gesture identifiers. At1004, if the user passes authentication, the processor may access aconfiguration file specifying associations for the library. At 1006, theprocessor may output a display of current registrations, such as a listof gesture identifiers and associated action identifiers, usinghuman-readable descriptions. At 1008, the processor may scan or searchthe configuration file to find a record requested by the user. At 1010,the processor may display a gesture returned by the search. In analternative, the processor may omit the search 1008 and display 1010 ifthe user does not specify any gesture.

At 1012, the processor may present the user with a menu, including atleast three possibilities: delete selected gesture, edit selectedgesture, or add new gesture. If the user selects “delete,” the processormay delete the gesture record at 1014, and at 1026, confirm the deletionand return to 1006 until user editing is finished.

If the user selects “edit,” the processor may enable user selection of anew action and/or target, at 1016. For example, the processor maypresent an interface enabling user selection of a target from targetsavailable to the user, and an action from available actions for eachtarget. At 1018 in response to a user selection, the processor mayreplace the prior action and/or target in the configuration record withthe newly selected action and/or target. Then the processor may confirmthe change at 1026 and revert to the registration display until the userediting is finished.

If the user selects “add new action” at 1012, the processor may define anew gesture at 1020, for example, using the method 600 described inconnection with FIG. 6 . At 1022, the processor may enable userselection of any available action and/or target, for example asdescribed in connection with block 1016. At 1026, the processor mayconfirm the change at 1026 and revert to 1006.

In accordance with an embodiment, as described above, the user maycreate corresponding gestures, which should not be construed to belimiting the scope of the disclosure. Notwithstanding, the disclosuremay not be so limited, and in accordance with another embodiment, thegestures may be pre-programmed and stored within a gesture recognitionsystem, that may execute within and/or in the background of anapplication of an external electronic device (such as a mobile device).

FIG. 11 shows a method 1100 for controlling a GR device to provide agesture-centric user interface for controlling one or more connecteddevices.

In accordance with the foregoing, and by way of additional example, FIG.11 shows more general aspects of a method or methods 1100 according toone embodiment, as may be performed by one or more processors of a GRdevice or system as described herein. It should be appreciated that themore general operations of method 1100 may include or embody moredetailed aspects of corresponding methods described herein above.

Referring to FIG. 11 , a computer-implemented method 1100 for providinga gesture-centric user interface for multiple target devices mayinclude, at 1110, sensing motion of a GR device comprising an inertialmotion sensor in three-dimensional space coupled to one or moreprocessors.

The method 1100 may further include, at 1120, matching a pattern of themotion to a gesture identifier. The method 1100 may further include, at1130, determining a target device and action identifier by reference toa data structure that associates each of a plurality of gestureidentifiers to a user-settable action identifier and target identifier.The method 1100 may further include, at 1140, requesting the targetdevice to perform an action identified by the action identifier

The method 1100 may include any one or more additional operations asdescribed herein above. Each of these additional operations is notnecessarily performed in every embodiment of the method, and thepresence of any one of the operations does not necessarily require thatany other of these additional operations also be performed. For example,optionally, method 1100 may further include a method 1000 of editing alibrary of gesture/action associations, or a method 600 for training aprocessor to recognize a gesture.

FIG. 12A is a diagram that illustrates an external view of an exemplaryGR device 1200, in accordance with an embodiment of the disclosure. Withreference to FIG. 12A, there is shown an external view 1200A of theexemplary GR device 1200, such as an interactive wand or a smart wand.The external view 1200A depicts an outer shell 1202 that comprises ahandle 1204 and a shaft 1206. The outer shell 1202 includes variousportions, such as an opaque portion 1202A, a transparent portion 1202B,and/or a translucent portion 1202C, that overall provides a classic,authentic, and dynamic appearance to the exemplary GR device 1200. Thehandle 1204 may be grasped by the hand of the user 99, while the shaft1206 provides a form factor of a wand tip extending outward from thehandle 1204. The outer shell 1202 may be structured in such a mannerthat a single printed circuit board assembly (PCBA) 1208 (as illustratedand described in FIGS. 12B and 12C) may be easily slid into the outershell 1202 and coupled thereto using a fastening mechanism, such aspress-and-fit clip.

The opaque portion 1202A, that may be made up of metal, wood, blendedpolymer or a combination thereof, spans the majority of the outer shell1202 of the exemplary GR device 1200 and masks the non-illuminatingcomponents of the PCBA 1208, and also adds to the aesthetic appeal ofthe exemplary GR device 1200. The transparent portion 1202B and thetranslucent portion 1202C provide an outer chassis to variousilluminating components, such as multi-colour LEDs, mounted on the PCBA1208 when the exemplary GR device 1200 is in an active state.

The transparent portion 1202B may be positioned preferably at the tip ofthe exemplary GR device 1200 to provide a bright illumination when theexemplary GR device 1200 is in the active state. On the other hand, thetranslucent portion 1202C may be positioned above the handle 1204, atthe bottom, and along the length of the shaft 1206 of the exemplary GRdevice 1200 to provide a diffused and subtle illumination when theexemplary GR device 1200 is in the active state.

The positions of the transparent portion 1201B and the translucentportion 1201C in the outer shell 1202 may correspond to the position ofthe underneath illuminating components mounted on the PCBA 1208. Thetransparent portion 1202B and the translucent portion 1202C mayilluminate in multi-colours when the user 99 activates the exemplary GRdevice 1200 and subsequently provides gestures to perform a specificaction, such as interacting with smart devices at home, franchiselocations, events, or bespoke enchanted items, thereby providing aspellcasting and magical experience to the user 99.

It may be noted that the form factor of the exemplary GR device 1200 inaccordance with the external view 1200A is provided merely for exemplarypurposes and should not be construed to limit the scope of thedisclosure. Notwithstanding, other suitable form factors of theexemplary GR device 1200 may be possible for being held, moved throughthe air, worn and/or carried by movements of the extremities of the user99. Each form factor of the exemplary GR device 1200 may define andprescribe a particular shape, size, pattern, material, and otherphysical specifications, without any deviation from the scope of thedisclosure.

In accordance with an embodiment, the exemplary GR device 1200 may be awireless interactive wand or a smart wand configured to communicatewirelessly via radio frequency (RF) or infrared (IR) communication modewith other devices by utilizing power generated by a power storage unit110. In accordance with another embodiment, the exemplary GR device 1200may be an interactive wand configured to illuminate at a plurality ofsections by utilizing power generated by a power storage unit 110. Inaccordance with yet another embodiment, the exemplary GR device 1200 maybe an interactive wand configured to generate haptic feedback byutilizing power generated by a power storage unit 110. For example, theexemplary GR device 1200 may light-up, vibrate and/or buzz at climacticmovements while watching a movie or a show. In accordance with yetanother embodiment, the exemplary GR device 1200 may communicate with anenchanted object, via an external electronic device (such as a mobiledevice), without use of a router. In certain embodiments, the externalelectronic device comprises a gesture recognition engine (not shown) forclassifying the gestures by type and associating the action identifiersand gesture types. In other embodiments, the exemplary GR device 1200may comprise the gesture recognition engine that may be executed by theexemplary GR device 1200 or via connection to the cloud.

FIGS. 12B and 12C are diagrams illustrating two views of an internalcircuitry of the exemplary GR device 1200 with a capacitive touch sensorpanel for detecting a multi-factor touch assertion, in accordance withan embodiment of the disclosure. With reference to FIGS. 12B and 12C,there is shown the single PCBA 1208 that includes the base unit 102 andthe elongate unit 104, as introduced in FIG. 1 . For the exemplary GRdevice 1200 styled as an interactive and/or a smart wand, the base unit102, upon which the handle 1204 is mounted, may be configured to begrasped by the hand of the user 99. Further, the elongate unit 104, uponwhich the shaft 1206 is mounted, may be configured to have a form factorof a wand tip that extends outward from the base unit 102 and providesan overall form factor to the wand. It may be noted that both the handle1204 and the shaft 1206 form the outer shell 1202 that is mounted as asingle unit on the single PCBA 1208. The single outer shell 1202 and thesingle PCBA 1208 are secured with each other using a locking feature,such as press-and-fit, to prevent undesired decoupling as the exemplaryGR device 1200 is rapidly moved around, for example while generichandling or while providing a gesture during spellcasting by the user99.

In some implementations, all the main electronic components of theexemplary GR device 1200, such as the controller 106, the power storageunit 110 (such as battery), the IMU 108, the RFID sensor, and infrared(IR) sensors, may be mounted on the base unit 102. On the other hand,the elongate unit 104 includes minimal number of electronic components,such as LEDs (for example LED 1210) with associated capacitors andregisters. Such a distribution of the electronic components frees up thespace on the elongate unit 104. Consequently, the elongate unit 104 isless likely to bend, distort, and flex around when the exemplary GRdevice 1200 is moved rapidly by the user 99. It may be noted that, forthe sake of brevity, such components are not shown in FIGS. 12B and 12C,as they have been already described in detail in FIG. 1 .

In addition to the aforesaid electronic components, FIGS. 12B and 12Cfurther illustrate two views, i.e. an unassembled view 1200B and anassembled view 1200C, of the internal circuitry, i.e. the single PCBA1208, of the exemplary GR device 1200, in accordance with an embodimentof the disclosure. The single PCBA 1208 further comprises a capacitivetouch sensor panel 1212 for detecting a multi-factor touch assertion.The capacitive touch sensor panel 1212 may include a plurality ofsensing pads, such as a first sensing pad 1212A, a second sensing pad1212B, a third sensing pad 1212C, and a fourth sensing pad 1212D, thatmay be arranged in a cylindrical pattern inside the handle 1204 of theexemplary GR device 1200. In an exemplary scenario, each of theplurality of sensing pads may be based on capacitive coupling that isrealized based on energy transfer within an electrical network (orbetween distant networks) by means of a displacement current betweennodes of a circuit nodes, induced by the electric field. Each of theplurality of sensing pads may be configured to detect anything that isconductive or has a dielectric different from air. For example, thehuman body (i.e. the digits of the fingers of the hand) may be used asan electrical charge conductor when the user 99 holds the exemplary GRdevice 1200 in accordance with a plurality of grip techniques, asdescribed in FIGS. 14A to 14D.

It may be noted that, based on experimental data, the number of sensingpads in the capacitive touch sensor panel 1212 are depicted anddescribed as four to offer optimum granularity at optimum cost. In a usecase, the number of sensing pads may be lesser than four. However,lesser number of sensing pads may offer more false positives thuslimiting the performance of the exemplary GR device 1200. In accordancewith an embodiment, the number of sensing pads may be greater than four,for example five sensing pads. In such an embodiment, the additionalsensing pads may be required to be supported optimally by additionalhardware, such as the controller 106 implemented as a Bluetoothsystem-on-chip (SoC) that incorporates Bluetooth Low-Energy (BLE) withan embedded Bluetooth radio, to increase the number of I/O units andsensing channels/lines.

In the context of the present disclosure, one or more digits or fingersof the hand of the user 99 placed near or on a set of sensing pads (fromthe plurality of sensing pads) in accordance with a specific griptechnique, asserts the set of sensing pads. The user 99 may place one ormore digits or fingers of the hand of the user 99 near or on a set ofsensing pads to hold the exemplary GR device 1200 in accordance with oneof a plurality of grip techniques. The plurality of grip techniques areillustrated and described in FIGS. 14A to 14D. Based on one of theplurality of grip techniques, corresponding detection factors aredetermined, and consequently, the assertion signal is generated thatcorresponds to one or more of the three detection factors. Suchmulti-factor touch assertion validates that the exemplary GR device 1200is ready and actually asserted, thus inferring the intent of the user 99to cast a spell, i.e., perform a gesture in the connected environment200 (as described in FIG. 2 ) based on a pre-defined gesture and/orvoice commands. In one example, using the exemplary GR device 1200, theuser 99 may interact with other IoT devices through an intermediarysystem such as, for example, Google Home®, Alexa®, or other IoT hub. Inanother example, the user 99 may control virtual objects in a AR or VRenvironment, and interact with real and/or virtual objects or projectedimages to unlock additional content and/or bonus features in anentertainment setting.

In accordance with an embodiment, as shown in FIGS. 12B and 12C, thearrangement of the cylindrical pattern of the plurality of sensing padsis such that a longitudinal axis of each sensing pad is arrangedorthogonally to a circular axis inside the handle 1204 of the exemplaryGR device 1200. As illustrated, the arrangement of the cylindricalpattern of the plurality of sensing pads enables a 360-degree capacitivetouch for the multi-factor touch assertion.

In accordance with an embodiment, the capacitive touch sensor panel 1212may be integrated on a flex printed circuit board 1214 and wrapped toform a cylindrical shape within the exemplary GR device 1200. The flexprinted circuit board 1214 may include printed or embedded circuits on acable plane thereby functioning as a flexible PCB. Furthermore, thecapacitive touch sensor panel 1212 may be communicatively coupled to thePCBA 1208 housing the controller 106 via a flex connector 1216. The flexconnector 1216 may be made of flexible plastic, polymers, films, orengineered rubber, with a metallic connector at the end, which may beembedded in parallel to the base.

In accordance with an embodiment, each of the plurality of sensing pads,such as the first sensing pad 1212A, the second sensing pad 1212B, thethird sensing pad 1212C, and the fourth sensing pad 1212D, may becommunicatively coupled to a capacitive touch sensor controller 1218.The capacitive touch sensor controller 1218 may be a dedicated controlunit for the capacitive touch sensor panel 1212 and may includecomponents, such as a processor, memory holding program instructions,and radio components, for controlling and managing at least theplurality of sensing pads. In an exemplary embodiment, the capacitivetouch sensor controller 1218 may be implemented as a Bluetoothsystem-on-chip (SoC) that incorporates Bluetooth Low-Energy (BLE) withan embedded Bluetooth radio. In accordance with one configuration, oneof the capacitive touch sensor controller 1218 or the controller 106, asa standalone control unit, may be configured to perform all theoperational steps, as described in FIG. 13 , without any deviation fromthe scope of the disclosure. In accordance with an alternateconfiguration, the capacitive touch sensor controller 1218 and thecontroller 106, in conjunction with each other as an integrated controlunit, may be configured to perform the operational steps, as describedin FIG. 13 , without any deviation from the scope of the disclosure. Inaccordance with another alternate configuration, the capacitive touchsensor controller 1218 the controller 106, and an external controller ofan external electronic device, though not shown here for the sake ofbrevity, in conjunction with each other as an integrated control unit,may be configured to perform the operational steps, as described in FIG.13 , without any deviation from the scope of the disclosure.

In a use case, while operating as the integrated control unit, thecapacitive touch sensor controller 1218 may serve as an ancillarycontrol unit to perform the initial simple operational steps oftransmitting driving signal to the plurality of sensing pads, activatingthe plurality of sensing pads, and detecting multi-factor touchassertion at a set of sensing pads from the activated plurality ofsensing pads. The complex operational steps, for example, generatingassertion signal, determining a signal sequence, inferring a userintent, converting a current inactive state of the GR device to anactive state based on a validation of the determined signal sequence,and communication of one or more signals to an external electronicdevice, such as a mobile device, via a communication channel (such asradio, cellular, or a wireless communication channel) may be performedby the controller 106, thus serving as the primary control unit.

In accordance with an embodiment, the first sensing pad 1212A, thesecond sensing pad 1212B, the third sensing pad 1212C, and the fourthsensing pad 1212D may be activated based on the driving signals receivedfrom the capacitive touch sensor controller 1218 or directly from thecontroller 106. When the user 99 holds the handle 1204 in accordancewith one of the plurality of grip techniques, as described in FIGS. 14Ato 14D, an electrical change at a set of sensing pads, for example twoopposing sensing pads, occurs that correspond to a two-factor touchassertion. The capacitive touch sensor controller 1218 or the controller106 may further determine the location of the touch indirectly from thechange in the capacitance as measured from the four corners of thecorresponding sensing pad. Within a pre-defined interval, the user 99may further provide a tap gesture or a 3D movement, that may be detectedby the IMU 108 and thus, provides a three-factor touch assertion. Thecapacitive touch sensor controller 1218 or the controller 106 may detectthe electrical change corresponding to the multi-factor touch assertionin response to an applied driving voltage. The capacitive touch sensorcontroller 1218 or the controller 106 may generate an assertion signalin accordance with the three detection factors corresponding to thedetected multi-factor touch assertion. In accordance with variousembodiments, the capacitive touch sensor controller 1218 or thecontroller 106 may be configured to perform further processing on thegenerated assertion signal.

It may be noted that, as described above and hereinafter, themulti-factor touch assertion realized based on capacitive touchtechnology (i.e. by the combination of the capacitive touch sensor panel1212 and the capacitive touch sensor controller 1218 (or the controller106)) is merely for exemplary purpose and should not be construed tolimit the disclosure. Notwithstanding, apart from the capacitive touchsensors, there may be other such means, such as pressure sensors,structured in the similar manner and performing the same functionalityof inferring the user intent from multi-factor touch assertion, withoutdeviating from the scope of the disclosure.

FIG. 12D is a circuit diagram 1200D of I/O drive of the capacitive touchsensor panel 1212 in the exemplary GR device 1200. With reference toFIG. 12D, there are shown four resistors 1220A, . . . , 1220D and fourcapacitors 1222A, . . . , 1222D connected with the first sensing pad1212A, the second sensing pad 1212B, the third sensing pad 1212C, andthe fourth sensing pad 1212D, respectively. Also shown are the drivingsignal, such as the capacitive touch driving signal 1224, and foursensing lines 1226A, . . . , 1226D.

The capacitive touch driving signal 1224 is transmitted to the firstsensing pad 1212A, the second sensing pad 1212B, the third sensing pad1212C, and the fourth sensing pad 1212D, via the four resistors 1220A, .. . , 1220D, for activating the capacitive touch sensor panel 1212. Forany assertion observed across a set of sensing pads, correspondingchange in the capacitance levels may be communicated via correspondingsensing lines from the sensing lines 1226A, . . . , 1226D, based onwhich assertion signal is generated by the capacitive touch sensorcontroller 1218 or the controller 106. Optionally, the four capacitors1222A, . . . , 1222D may be connected across the four resistors 1220A, .. . , 1220D respectively for electromagnetic interference (EMI)filtering to filter out unwanted noise signals. It may be noted that forthe sake of simplicity, the circuit diagram 1200D is illustrated hereinto implement a two-factor touch assertion and without showing anassertion by the third factor.

FIG. 13 is a flow chart 1300 illustrating a method for sensingmulti-factor touch assertion by the exemplary GR device 1200. FIG. 13 isdescribed in conjunction with FIGS. 14A to 14D, that are use casediagrams of the exemplary GR device 1200 illustrating various griptechniques for the multi-factor touch assertion, in accordance withvarious embodiments of the disclosure.

At 1302, a driving signal may be transmitted to each of the plurality ofsensing pads for the detection of the multi-factor touch assertion. Inaccordance with an embodiment, the control unit, such as the capacitivetouch sensor controller 1218 or the controller 106, may be configured totransmit the driving signal to each of the plurality of sensing pads,such as the first sensing pad 1212A, the second sensing pad 1212B, thethird sensing pad 1212C, and the fourth sensing pad 1212D, for thedetection of the multi-factor touch assertion.

In accordance with an embodiment, the controller 106 or the capacitivetouch sensor controller 1218 may sequentially supply the driving signal,having a specific frequency, to the capacitive touch sensor panel 1212.The value of the specific frequency of the driving signal received fromthe controller 106 or the or the capacitive touch sensor controller 1218has a pre-defined value. Accordingly, the capacitive touch sensor panel1212 may be driven for detecting multiple touch inputs provided by theuser 99. The driving signal applied to the capacitive touch sensor panel1212 may include a plurality of driving pulses. For example, the drivingsignal may be a square wave operated from an integrated circuit's (IC's)supply voltage, e.g., 3.0V. In certain embodiments, to minimize theinfluence of noise in the driving pulse, the width of the driving pulsemay be changed while maintaining the frequency of the driving signal. Insuch embodiments, the width of the driving pulse may be changed when thenoise level exceeds a threshold value.

At 1304, a plurality of sensing pads may be activated based on thedriving signal. In accordance with an embodiment, the control unit, suchas the capacitive touch sensor controller 1218 or the controller 106,may be configured to activate the plurality of sensing pads of thecapacitive touch sensor panel 1212 based on the received driving signal.

Each of the plurality of sensing pads of the capacitive touch sensorpanel 1212 has a touch-sensing surface that uses an array of touchsensors to detect assertions on the surface corresponding to the touchinputs provided by the user 99. In accordance with an embodiment, eachtouch sensor in the array of touch sensors may receive the drivingsignals and may be coupled with sensing lines on a one-on-one basisthrough additional electronic components, i.e. resistor and a set ofcapacitors. The control unit, such as the capacitive touch sensorcontroller 1218 or the controller 106, may be configured to continuouslymeasure the self-capacitance of each sensing pad in the capacitive touchsensor panel 1212 in the activated state.

At 1306, a multi-factor touch assertion may be detected at the set ofsensing pads from the activated plurality of sensing pads. In accordancewith an embodiment, the set of sensing pads may be configured to detectthe multi-factor touch assertion from the activated plurality of sensingpads. The multi-factor touch assertion may be detected from theactivated set of sensing pads when the self-capacitance of an electrodeof the touch-sensing surface of corresponding sensing pad changes. Sucha change occurs when a conductive body or material, for example handdigits of the user 99, having dielectric different from air is placednear or on the corresponding sensing pad.

In accordance with various use cases, the self-capacitance of theelectrode of the touch-sensing surface of corresponding sensing padchanges when the user 99 holds the handle 1204 of the exemplary GRdevice 1200 through various gripping techniques 1400A to 1400D, asillustrated in FIGS. 14A to 14D.

In a first example use case diagram, as illustrated in FIG. 14A, thereis depicted a first grip technique 1400A for the multi-factor touchassertion. As shown, for holding the exemplary GR device 1200, the firstgrip technique 1400A corresponds to a first touch and a continuallymaintained assertion by a second digit 1404 of hand, i.e. the indexfinger, on at least one sensing pad, such as the first sensing pad1212A, when the index finger is underneath the handle 1204. By way ofsuch technique the thumb may not touch the opposite sensing pad and thehandle 1204 appears to be resting entirely on the second digit. However,more than one sensing pads may also be asserted by the index fingerbased on placement of the hand of the user 99 on the handle 1204. Forexample, if the grip is substantially tight, the index finger mayencompass up to three sensing pads, as shown in FIG. 14A. In suchexample, the fourth sensing pad may be asserted by the perlicue, i.e.the skin portion between the base on thumb and index finger whenextended.

In a second example use case diagram, as illustrated in FIG. 14B, thereis depicted a second grip technique 1400B for the multi-factor touchassertion. As shown, for holding the exemplary GR device 1200, thesecond grip technique 1400B corresponds to a first touch and secondtouch continually maintained assertions by the first digit 1402, i.e.the thumb, and a third digit 1406 of hand, i.e. the middle finger, ontwo or more sensing pads, such as the first and the third sensing pads1212A and 1212C, respectively, and a third touch asserted by the seconddigit 1404 of the hand, i.e. the index finger, on a remaining sensingpad, such as the second sensing pad 1212B, between the first sensing pad1212A and the third sensing pad 1212C.

In a third example use case diagram, as illustrated in FIG. 14C, thereis depicted a third grip technique 1400C for the multi-factor touchassertion. As shown, for holding the exemplary GR device 1200, the thirdgrip technique 1400C for the multi-factor touch assertion corresponds tothe first touch and maintained assertion by a first digit 1402 of hand,i.e. the thumb, on at least one sensing pad, such as the first sensingpad 1212A, and a second touch asserted by the second digit 1404 of thehand, i.e. the index finger, on at least one remaining sensing pad, suchas the third sensing pad 1212C, positionally opposite to the firstsensing pad 1212A.

In a fourth example use case diagram, as illustrated in FIG. 14D, thereis depicted a fourth grip technique 1400D for the multi-factor touchassertion. As shown, for holding the exemplary GR device 1200, thefourth grip technique for the multi-factor touch assertion correspondsto a tap gesture provided by the first digit 1402 or the second digit1404 of hand on at least one sensing pad. The tap gesture may bedetected by the accelerometers 172 in the IMU 108, coupled with theplurality of sensing pads of the capacitive touch sensor panel 1212. Theaccelerometers 172 may detect the tap gesture by movement of the firstdigit 1402 from a first position 1401A at timestamp T1 on the surface ofthe handle 1204, to a second position 1401B of the first digit attimestamp T2 away from the surface of the handle 1204, and then back tothe first position 1401A at timestamp T3 on the surface of the handle1204, as illustrated in FIG. 14D. Such a movement is detected as the tapgesture if the two times change in positions, i.e. from position 1401Ato position 1401B, and then back to position 1401A, spans betweentimestamps T1 to T3 such that the time interval is less than apre-defined threshold time interval value.

At 1308, an assertion signal may be generated in response to thedetected multi-factor touch assertion. In accordance with an embodiment,the capacitive touch sensor controller 1218 may be configured togenerate the assertion signal in response to the detected multi-factortouch assertion and transmit the generated assertion signal to thecontroller 106. The controller 106 may be configured to receive thegenerated assertion signal from the capacitive touch sensor controller1218 for further processing of the assertion signal. In accordance withanother embodiment, the controller 106 may be configured to directlygenerate the assertion signal in response to the detected multi-factortouch assertion.

As the multi-factor touch assertion is detected based on a change in thedriving signal voltage before or after a touch or a rising or fallingedge delay time of the driving signal to sense a change in capacitancebefore or after the touch (or a proximity touch) is provided. Thecontroller 106 and/or the capacitive touch sensor controller 1218converts a voltage received from touch (capacitive) sensors of thetouch-sensing surface into digital data to generate raw data. Inaccordance with an embodiment, the controller 106 and/or the capacitivetouch sensor controller 1218 may analyze the raw data based on apre-stored touch recognition algorithm to detect the touch (or aproximity touch) input and generate a corresponding assertion signal.

In accordance with various embodiments, by using the exemplary GR device1200, the assertion signal may be generated based on the detectedmulti-factor touch assertion for various grip techniques 1400A to 1400D,as illustrated in FIGS. 14A to 14D, as described in detail above. Inaccordance with different grip techniques, the multiple detectionfactors may be identified. The multiple detection factors may include afirst detection factor, a second detection factor and an optional thirddetection factor.

The first detection factor of the multi-factor touch assertion may be aninitial assertion of at least one sensing pad, for example the firstsensing pad 1212A, on one side of the handle 1204 resulting from anatural or learned grip technique performed by the user 99. For example,the first detection factor may be identified corresponding to the firstgripping technique 1400A.

The second detection factor may be a subsequent additional assertion ofat least one opposite facing sensing pads, for example the first sensingpad 1212A and the third sensing pad 1212C, by the thumb or index fingerof the user 99. For example, the second detection factor may beidentified corresponding to the second and the third gripping technique1400B and 1400C, respectively. In accordance with an embodiment, thesecond detection factor may be enhanced by monitoring and reacting tothe amount of capacitance detected at the opposite facing (or remaining)sensing pads. Accordingly, the controller 106 may be configured todistinguish between the second detection factor and the first detectionfactor in an exemplary scenario when the finger (or the digit) of theuser 99 overlaps with another pad(s) during the initial assertion of theat least one sensing pad (in accordance with the first detectionfactor). For example, in accordance the exemplary scenario, during theinitial assertion, the index finger of the user 99 may partially overlapwith the adjacent sensing pad (with a detected capacitance “c”) whilecompletely asserting the first sensing pad 1212A (with a detectedcapacitance “C1”). Thus, the amount of capacitance detected by thecontroller 106 may be “C1+c” as the first sensing pad 1212A iscompletely asserted and the adjacent sensing pad is partially asserted.On the other hand, during the subsequent assertion, the index finger ofthe user 99 completely asserts the first sensing pad 1212A (with thedetected capacitance “C1”) and the thumb completely asserts the oppositesensing pad, i.e. the third sensing pad 1212C (with a detectedcapacitance “C3”). Thus, the amount of capacitance detected by thecontroller 106 may be “C1+C3” as two opposite sensing pads arecompletely asserted. Clearly, the amount of detected capacitancecorresponding to the second detection factor is more than the firstdetection factor, based on which the controller 106 may distinguishbetween the two detection factors in case of such exemplary scenario.

The third optional detection factor may be sensing a three dimensionalmovement of the exemplary GR device 1200 during the subsequentadditional assertion. An example of the third optional detection factormay correspond to a waving gesture performed by the user 99 holding theexemplary GR device 1200 during the subsequent additional assertion.Another example of the third optional detection factor may correspond toa tap gesture that may be detected by the accelerometers 172 in the IMU108, in accordance with the fourth gripping technique 1400D. The thirdoptional detection factor may be considered mandatory in all thegripping techniques 1400A to 1400D to further validate the user intentto use the exemplary GR device 1200 to minimize or exclude any possibleoccurrence of false positives.

In accordance with an embodiment, the assertion signal may be generatedbased on the detected multi-factor touch assertion for the plurality ofgrip techniques 1400A to 1400D. In accordance with an embodiment, theassertion signal generated by the capacitive touch sensor controller1218 or the controller 106 may be further communicated to an externalelectronic device, such as a mobile device (for example a cell phone ora computing device on which corresponding application program isinstalled), via a communication channel. In such embodiment, thefollowing steps 1310 to 1316 may be performed by an external controllerof external electronic device, not shown for the sake of simplicity.

At 1310, a signal sequence may be determined based on the generatedassertion signal. In accordance with an embodiment, the controller 106may be configured to determine the signal sequence based on thegenerated assertion signal. In accordance with an embodiment, thecontroller 106 may be configured to communicate the generated assertionsignal to the external controller, and the external controller maydetermine the signal sequence based on the received assertion signal.The controller 106 (or the external controller) may be configured tovalidate the determined signal sequence corresponding to themulti-factor touch assertion. Such validation may correspond to timestamps of the assertions of at least one of the plurality of sensingpads, optionally followed by an input from the IMU 108 (in accordancewith the third detection factor) to better determine the user intent ona press gesture provided on the handle 1204 of the GR device 1200.

The controller 106 (or the external controller) may validate thedetermined signal sequence if the user 99 is holding the exemplary GRdevice 1200 based on one of the grip techniques as described in FIGS.14A to 14D. For example, in accordance with the second grippingtechnique 1400B, a valid signal sequence may be if the user 99 holds theexemplary GR device 1200 by placing the third digit 1406 of hand, i.e.the middle finger, on the first sensing pad 1212A, and the first digit1402 of the hand, i.e. the thumb, the third sensing pad 1212C,positionally opposite to the first sensing pad 1212A. Further, thesecond digit 1404 of hand, i.e. the index finger, is placed on thesecond sensing pad 1212B. Thus, a sequence of assertion signalscorresponding to the first detection factor (due to initial assertion ofthe first sensing pad 1212A on one side of the handle 1204 resultingfrom a natural or learned grip technique performed by the user 99), thesecond detection factor (due to subsequent additional assertion of thepositionally opposite third sensing pad 1212C by the thumb) and thethird detection factor (due to sensing of the tap gesture or the threedimensional movement of the GR device 1200 during the subsequentadditional assertion and within a time interval less than thepre-defined threshold time interval value) may correspond to a validsignal sequence.

On the contrary, the user 99 may hold the exemplary GR device 1200 insuch a manner that all of the plurality of sensing pads of thecapacitive touch sensor panel 1212 are asserted at once for a timeinterval that exceeds the pre-defined threshold time interval value.Further, the timestamp of such concurrent assertions are not separatedas all of the plurality of sensing pads are asserted at the same time.Furthermore, the IMU 108 is not activated due to absence of any movementof the exemplary GR device 1200. In such case, the controller 106 (orthe external controller) may register the assertion as a false positive.Accordingly, the controller 106 (or the external controller) may discardsuch assertion as being false positive. Thus, in accordance with a validsignal sequence, the controller 106 (or the external controller) may beconfigured to detect the first and the second detection factors followedby the third detection factor to exclude false positives and infer theuser intent to use the exemplary GR device 1200 in a productive manner.

In accordance with an embodiment, there may be a possibility that theuser 99 is unintentionally providing a first touch input and asubsequent second touch input by using only two digits, however, theplacement of the digits is such that eventually all the four sensingpads are asserted. For example, as identified as the first detectionfactor, the user 99 may hold the exemplary GR device 1200 such that thesecond digit 1404 of hand, i.e. the index finger, is placed at a commonedge between the first sensing pad 1212A and the second sensing pad1212B, such that both the first sensing pad 1212A and the second sensingpad 1212B are asserted at once. In such case, as identified as thesecond detection factor, as a subsequent assertion, if the user placesthe first digit 1402 of the hand, i.e. the thumb, on the opposite side,i.e. at the common edge between the third sensing pad 1212C and thefourth sensing pad 1212D, the capacitive touch sensor the controller 106(or the external controller) may not validate such signal sequence.

Accordingly, a sequence of assertions, i.e. the initial pair and/orsubsequent pair of assertions at opposite sides must be followed by thethird detection factor that corresponds to sensing a three dimensionalmovement of the GR device 1200 during the subsequent pair of assertionwithin a time interval that is less than the pre-defined threshold timeinterval value, as detected by the IMU 108 (for example, theaccelerometers 172). Accordingly, the controller 106 (or the externalcontroller) may validate the signal sequence and exclude any possibilityof registering such signal sequence as a false positive.

At 1312, a user intent may be inferred based on the detectedmulti-factor touch assertion and determined signal sequence. Inaccordance with an embodiment, the controller 106 (or the externalcontroller) may be configured to infer the user intent based on thedetected multi-factor touch assertion and determined signal sequence.The user intent may be inferred based on a combination of type of griptechnique on the handle 1204 and subsequent finger and/or thumb pressperformed by the user 99, and also the time sequence of assertions. Anexample of inferring the user intent may be based on how the user 99holds the exemplary GR device 1200, such as a smart and interactivewand, based on the one or more factors and how the user 99 performs agesture, for example press and hold the wand with the thumb once or movethe wand around in the air, to cast a spell.

At 1314, a current inactive state of the exemplary GR device 1200 may beconverted to an active state based on a validation of the determinedsignal sequence corresponding to the multi-factor touch assertion andinferred user intent. In accordance with an embodiment, the controller106 may be configured to convert the current inactive state of theexemplary GR device 1200 may be converted to an active state based onthe validation of the determined signal sequence corresponding to themulti-factor touch assertion and inferred user intent, as discussedabove. The conversion of the current state of the exemplary GR device1200 to the active state may be further based on the driving signalreceived from the controller 106 and a touch gesture received from atleast a digit of hand of the user 99. Once in the active state, theexemplary GR device 1200 may be used in the context of the connectedenvironment 200, as described in detail in FIG. 2 . For example, theactivated GR device 1200 may be configured to control other connecteddevices based on gestures and/or voice commands of the user 99, controlvirtual objects in an AR or VR environment, interact with real and/orvirtual objects or projected images, unlock additional content and/orbonus features in an entertainment setting, control the narrative of amovie, or identify an item selected by a customer using gesture controlin a retail store.

After step 1314, the control passes to step 1306 to detect themulti-factor touch assertion from the activated plurality of sensingpads. In an embodiment when the multi-factor touch assertion isdetected, the control passes to step 1308, and the process continues, asdescribed above. However, in another embodiment when the multi-factortouch assertion is not detected, the control passes to step 1316.

At 1316, the active state of the exemplary GR device 1200 may beconverted to a sleep state in absence of the multi-factor touchassertion at the set of sensing pads for a pre-defined time duration. Inaccordance with an embodiment, the controller 106 (or the externalcontroller) may be configured to convert the active state of theexemplary GR device 1200 to the sleep state in absence of themulti-factor touch assertion at the set of sensing pads for apre-defined time duration.

The state of the exemplary GR device 1200, which is currently in thesleep state, may be changed back to the active state based on variousmechanisms. In accordance with a first mechanism, when there is anymovement, for example lifting, detected by one or more components in theIMU 108. In accordance with a second mechanism, for each of theplurality of capacitive touch sensors in the capacitive touch sensorpanel 1212, the sensing period and the duty cycle may be reduced to avalue less than a threshold value. In other words, the less frequenttouch sensing period and slower duty cycles of the capacitive touchsensors may be implemented that eventually reduces the power consumptionof the exemplary GR device 1200.

FIG. 15 shows components of an apparatus or system 1500 for providing agesture-centric user interface for multiple target devices and sensingmulti-factor touch assertion as described herein, according to methods1100 and 1300. As depicted, the apparatus or system 1500 may includefunctional blocks that can represent functions implemented by aprocessor, software, or combination thereof (e.g., firmware).

The apparatus or system 1500 may comprise an electrical component 1502for sensing motion of a GR device comprising an inertial motion sensorin three-dimensional space coupled to one or more processors. Thecomponent 1502 may be, or may include, a means for said sensing. Saidmeans may include the processor 1510 coupled to the memory 1516, and tothe inertial motion sensor 14, the processor executing an algorithmbased on program instructions stored in the memory. Such algorithm mayinclude a sequence of more detailed operations, for example, method 700as described in connection with FIG. 7 .

The apparatus or system 1500 may further comprise an electricalcomponent 1503 for matching a pattern of the motion to a gestureidentifier, e.g., recognizing the gesture. The component 1503 may be, ormay include, a means for said matching or recognizing. Said means mayinclude the processor 1510 coupled to the memory 1516, the processorexecuting an algorithm based on program instructions stored in thememory. Such algorithm may include a sequence of more detailedoperations, for example, method 804 as described in connection with FIG.8B.

The apparatus or system 1500 may further comprise an electricalcomponent 1504 for determining a target device and action identifier byreference to a data structure that associates each of a plurality ofgesture identifiers to a user-settable action identifier and targetidentifier. The component 1504 may be, or may include, a means for saidmatching. Said means may include the processor 1510 coupled to thememory 1516 containing a library data structure, the processor executingan algorithm based on program instructions stored in the memory. Suchalgorithm may include a sequence of more detailed operations, forexample, the method 800 described in connection with FIG. 8A.

The apparatus or system 1500 may further comprise an electricalcomponent 1505 for sensing multi-factor touch assertion. The component1505 may be, or may include, a means for said sensing. Said means mayinclude the processor 1510 coupled to the memory 1516 containing alibrary data structure, the processor executing an algorithm based onprogram instructions stored in the memory. Such algorithm may include asequence of more detailed operations, for example, the method 1300described in connection with FIG. 13 .

The apparatus or system 1500 may further comprise an electricalcomponent 1506 for requesting the target device to perform an actionidentified by the action identifier. The component 1506 may be, or mayinclude, a means for said requesting. Said means may include theprocessor 1510 coupled to the memory 1516, and to a network interfacedevice, the processor executing an algorithm based on programinstructions stored in the memory. Such algorithm may include a sequenceof more detailed operations, for example, the method 900 described inconnection with FIG. 9 .

The apparatus 1500 may optionally include a processor module 1510 havingat least one processor, in the case of the apparatus 1500 configured asa data processor. The processor 1510, in such case, may be in operativecommunication with the modules 1502-1506 via a bus 1512 or othercommunication coupling, for example, a network. The processor 1510 mayinitiate and schedule the processes or functions performed by electricalcomponents 1502-1506. The electrical components 1502-1506 may also bereferred to as circuits or circuitry.

In related aspects, the apparatus 1500 may include a network interfacemodule (not shown in FIG. 15 , shown in FIG. 1 ) operable forcommunicating with a targeted clients and network resources over acomputer network. In further related aspects, the apparatus 1500 mayoptionally include a module for storing information, such as, forexample, a memory device/module 1516. The computer readable medium orthe memory module 1516 may be operatively coupled to the othercomponents of the apparatus 1500 via the bus 1512 or the like. Thememory module 1516 may be adapted to store computer readableinstructions and data for effecting the processes and behavior of themodules 1502-1506, and subcomponents thereof, or the processor 1510, orthe any method or process described herein. The memory module 1516 mayretain instructions for executing functions associated with the modules1502-1506. While shown as being external to the memory 1516, it is to beunderstood that the modules 1502-1506 can exist within the memory 1516.

The proposed exemplary GR device 1200 and the method provides variousadvantages, such as a miniaturized form factor with low cost as comparedto the existing solutions. The multi-factor touch assertion providesanother advantage is that it is less likely to register false positives.Further, due to less frequent touch sensing period and slower dutycycles of the capacitive touch sensors, power consumption of theexemplary GR device 1200 may be reduced to a substantial extent ascompared to existing solutions.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the aspects disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the application and design constraints imposed on theoverall system. Skilled artisans may implement the describedfunctionality in varying ways for each application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

As used in this application, the terms “component”, “module”, “system”,and the like are intended to refer to a computer-related entity, eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component may be, but is not limited tobeing, a process running on a processor, a processor, an object, anexecutable, a thread of execution, a program, and/or a computer orsystem of cooperating computers. By way of illustration, both anapplication running on a server and the server can be a component. Oneor more components may reside within a process and/or thread ofexecution and a component may be localized on one computer and/ordistributed between two or more computers.

Various aspects will be presented in terms of systems that may includeseveral components, modules, and the like. It is to be understood andappreciated that the various systems may include additional components,modules, etc. and/or may not include all the components, modules, etc.discussed in connection with the figures. A combination of theseapproaches may also be used. The various aspects disclosed herein can beperformed on electrical devices including devices that utilize touchscreen display technologies and/or mouse-and-keyboard type interfaces.Examples of such devices include computers (desktop and mobile), smartphones, personal digital assistants (PDAs), and other electronic devicesboth wired and wireless.

In addition, the various illustrative logical blocks, modules, andcircuits described in connection with the aspects disclosed herein maybe implemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general-purpose processor may be a microprocessor,but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

Operational aspects disclosed herein may be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumknown in the art. An exemplary storage medium is coupled to theprocessor such the processor can read information from, and writeinformation to, the storage medium. In the alternative, the storagemedium may be integral to the processor. The processor and the storagemedium may reside in an ASIC. The ASIC may reside in a user terminal. Inthe alternative, the processor and the storage medium may reside asdiscrete components in a user terminal.

Furthermore, the one or more versions may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedaspects. Non-transitory computer readable media can include but are notlimited to magnetic storage devices (e.g., hard disk, floppy disk,magnetic strips . . . ), optical disks (e.g., compact disk (CD), digitalversatile disk (DVD), BluRay™ . . . ), smart cards, solid-state devices(SSDs), and flash memory devices (e.g., card, stick). Of course, thoseskilled in the art will recognize many modifications may be made to thisconfiguration without departing from the scope of the disclosed aspects.

The previous description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these aspects will be clear to those skilled inthe art, and the generic principles defined herein may be applied toother embodiments without departing from the spirit or scope of thedisclosure. Thus, the present disclosure is not intended to be limitedto the embodiments shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

In view of the exemplary systems described supra, methodologies that maybe implemented in accordance with the disclosed subject matter have beendescribed with reference to several flow diagrams. While for purposes ofsimplicity of explanation, the methodologies are shown and described asa series of blocks, it is to be understood and appreciated that theclaimed subject matter is not limited by the order of the blocks, assome blocks may occur in different orders and/or concurrently with otherblocks from what is depicted and described herein. Moreover, not allillustrated blocks may be required to implement the methodologiesdescribed herein. Additionally, it should be further appreciated thatthe methodologies disclosed herein are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethodologies to computers.

What is claimed is:
 1. A gesture recognition (GR) device, comprising: acapacitive touch sensor panel comprising a plurality of sensing padsarranged in a cylindrical pattern inside a handle of the GR device, thecapacitive touch sensor panel configured to: detect a multi-factor touchassertion at a set of sensing pads of the plurality of sensing pads; anda controller coupled to the plurality of sensing pads, the controllerconfigured to: transmit a driving signal to each of the plurality ofsensing pads for the detection of the multi-factor touch assertion;generate an assertion signal that corresponds to the detectedmulti-factor touch assertion, wherein the assertion signal is generatedbased on the detected multi-factor touch assertion for a plurality ofgrip techniques; determine a signal sequence based on the assertionsignal; and convert a current inactive state of the GR device to anactive state based on a validation of the determined signal sequencecorresponding to the multi-factor touch assertion and an inferred userintent.
 2. The GR device according to claim 1, wherein the arrangementof the cylindrical pattern of the plurality of sensing pads is such thata longitudinal axis of each sensing pad is arranged orthogonally to acircular axis inside the handle of the GR device.
 3. The GR deviceaccording to claim 1, wherein the arrangement of the cylindrical patternof the plurality of sensing pads enables a 360-degree capacitive touchfor the multi-factor touch assertion.
 4. The GR device according toclaim 1, wherein the controller is further configured to determine theinferred user intent based on the detected multi-factor touch assertionand the determined signal sequence, and wherein the inferred user intentis determined based on a combination of type of grip technique on thehandle and subsequent finger and/or thumb press performed by a user. 5.The GR drive according to claim 1, wherein the assertion signalcorresponds to one of a first detection factor, a second detectionfactor or an optional third detection factor, wherein the firstdetection factor is an initial assertion of at least one sensing pad onone side of the handle resulting from a natural or learned griptechnique performed by a user, wherein the second detection factor is asubsequent additional assertion of at least one positionally oppositesensing pad by thumb or index finger of the user, and wherein theoptional third detection factor is sensing a three dimensional movementof the GR device during the subsequent additional assertion.
 6. The GRdevice according to claim 5, wherein a first grip technique from theplurality of grip techniques for the multi-factor touch assertioncorresponds to a first touch and continually maintained assertion by asecond digit of hand on at least one sensing pad, and wherein theassertion signal is generated based on the first grip technique.
 7. TheGR device according to claim 5, wherein a second grip technique from theplurality of grip techniques for the multi-factor touch assertioncorresponds to a first touch and a second touch continually maintainedassertions by a first digit and a third digit of hand, respectively, onat least two sensing pads, and a third touch asserted by a second digitof the hand on a remaining sensing pad between the at least two sensingpads, and wherein the assertion signal is generated based on the secondgrip technique.
 8. The GR device according to claim 5, wherein a thirdgrip technique from the plurality of grip techniques for themulti-factor touch assertion corresponds to a first touch andcontinually maintained assertion by a first digit of hand on at leastone sensing pad and a second touch asserted by a second digit of thehand on the at least one positionally opposite sensing pad, and whereinthe assertion signal is generated based on the third grip technique. 9.The GR device according to claim 5, wherein a fourth grip technique fromthe plurality of grip techniques for the multi-factor touch assertioncorresponds to a tap gesture provided by a first or a second digit ofhand on at least one sensing pad, and wherein the assertion signal isgenerated based on the fourth grip technique.
 10. The GR deviceaccording to claim 9, wherein an accelerometer, coupled with theplurality of sensing pads, is configured to detect the tap gesture. 11.The GR device according to claim 1, wherein the controller is furtherconfigured to change the active state of the GR device to a sleep statein absence of the multi-factor touch assertion at the set of sensingpads for a pre-defined time duration.
 12. The GR device according toclaim 1, wherein the conversion of the current inactive state of the GRdevice to the active state is further based on the driving signalreceived from the controller and a touch gesture received from at leasta digit of hand of a user.
 13. The GR device according to claim 1,wherein a frequency value of the driving signal received from thecontroller has a pre-defined value.
 14. The GR device according to claim1, wherein a sensing period and a duty cycle of the plurality of sensingpads is less than a threshold value.
 15. The GR device according toclaim 1, wherein the GR device is a wireless interactive wand or a smartwand configured to communicate wirelessly via a radio frequency (RF) oran infrared (IR) communication mode with other devices by utilizingpower generated by a power storage unit.
 16. The GR device according toclaim 1, wherein the GR device is an interactive wand configured toilluminate in a plurality of sections by utilizing power generated by apower storage unit.
 17. The GR device according to claim 1, wherein theGR device is an interactive wand configured to generate haptic feedbackby utilizing power generated by a power storage unit.
 18. The GR deviceaccording to claim 1, wherein the capacitive touch sensor panel isintegrated on flex printed circuit board and wrapped to form acylindrical shape within the GR device.
 19. The GR device according toclaim 1, wherein the capacitive touch sensor panel is communicativelycoupled to a printed circuit board housing the controller via a flexconnector.
 20. A capacitive touch sensor panel, comprising: a pluralityof sensing pads arranged in a cylindrical pattern inside a handle of agesture recognition (GR) device, the arrangement of the plurality ofsensing pads in the cylindrical pattern is such that a longitudinal axisof each sensing pad is arranged orthogonally to a circular axis aroundthe handle of the GR device, the plurality of sensing padscommunicatively coupled to a capacitive touch sensor controller, thecapacitive touch sensor controller being configured to: transmit adriving signal to each of the plurality of sensing pads; detect amulti-factor touch assertion at a set of sensing pads from the pluralityof sensing pads; generate an assertion signal in response to thedetected multi-factor touch assertion, wherein the assertion signal isgenerated based on the detected multi-factor touch assertion for aplurality of grip techniques; and transmit the generated assertionsignal to a controller, wherein the GR device is activated by thecontroller based on a validation of a signal sequence determined basedon the generated assertion signal and an inferred user intent.
 21. Thecapacitive touch sensor panel according to claim 20, wherein theplurality of sensing pads is arranged in the cylindrical pattern insidethe handle such that the plurality of sensing pads is in proximity tohand digits of a user.
 22. A method for sensing multi-factor touchassertion, comprising: transmitting, by a control unit, a driving signalto a plurality of sensing pads for detection of multi-factor touchassertion; activating, by the control unit, the plurality of sensingpads based on the driving signal; detecting, by the control unit, themulti-factor touch assertion at a set of sensing pads from the activatedplurality of sensing pads; generating, by the control unit, an assertionsignal in response to the detected multi-factor touch assertion, whereinthe assertion signal is generated based on the detected multi-factortouch assertion for a plurality of grip techniques; determining, by thecontrol unit, a signal sequence based on the assertion signal;inferring, by the control unit, a user intent based at least on thedetected multi-factor touch assertion and the determined signalsequence; and converting, by the control unit, a current inactive stateof a gesture recognition device to an active state based on a validationof the determined signal sequence corresponding to the multi-factortouch assertion and the inferred user intent.
 23. The method accordingto claim 22, wherein the assertion signal is generated based on one of afirst grip technique, a second grip technique, a third grip technique,or a fourth grip technique from the plurality of grip techniques.